DOCUMENT RESUME 



JC 920 074 

Slark, Julie; And Others 

RSC Validation of Mathematics Placement Tests. 
Research, Planning, Resource Development Report. 
Rancho Santiago Community Coll., Santa Ana, Calif. 
Dec 91 
19p. 

Reports - Research/Technical (143) 
MF01/PC01 Plus Postage. 

Community Colleges; Comparative Analysis; Educational 
Testing; Grade Prediction; Grades (Scholastic); 
Mathematics; "Mathematics Achievement; "Mathematics 
Tests; "Predictive Validity; student Characteristics; 
"Student Placement; "Test Validity; Two Year 
Colleges; Two Year College students 



In an effort to determine the validity of placement 
tests used at Rancho Santiago college (Santa Ana, California), for 
placing students into mathematics courses, a study was conducted 
comparing the course performance of students who met test cut-off 
scores ("eligible" to take the course) with students who did not meet 
the cut-off scores ("ineligible"), students were tested using the 
California State University/University of California Mathematics 
Diagnostic Testing Project tests, and comparisons were made of the 
cours3 success rates for eligible and ineligible students in 
Elementary Algebra (Math 50) , Elementary Algebra Review (Math 60) , 
and Intermediate Algebra (Math 80) in fall 1990 and spring 1991. 
Successful students were defined as those receiving grades of A, B, 
or C, or receiving course credit, study results were as follows: (1) 
of the 454 students in Math 50, 83% scored above the placement test 
cut-off score, and 47% were successful in the course; (2) of the 
students in Math 50, 48% of those scoring above the cut-off scores 
were successful as compared with 41% of those scoring below the 
cut-off scores; (3) of the 266 students in Math 60, 94% scored above 
the cut-off and 61% ware successful in the course; (4) of the 
students in Math fiO, 62% of those scoring above the cut-off scores 
were successful in the course, compared with 47% of those scoring 
below the cut-off scores; (5) of the students in Math 80, 63% scored 
above the cut-off score and 63% were also successful in the course; 
(6) of the students in Math 80, 65% of those scoring above the 
placement test cut-off scores were successful in the course as 
compared with 58% of those scoring below the cut-off scores; and (7) 
correlations of placement test scores with course outcomes revealed 
the test cut-off scores were appropriate and should not be modified. 
Detailed data tables are included. (PAA) 
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Validation of Mathematics Placement Tests 



Design 

Students who had been placed as a result of an assessment test score in Elementary 
Algebra (Math 050), Elementary Algebra Review (060), and Intermediate Algebra (Math 080) 
were the subjects of this predictive validity study. Students were tested using the CSU/UC 
Mathematics Diagnostic Testing Project (MDTP) battery of tests. For purposes of this study, 
Test #1 refers to the MDTP Algebra Readiness test; Test #2 refers to the MDTP Elementary 
Algebra test; and Test #3 refers to the MDTP Intermediate Algebra test. In keeping with 
compliance with the State of California, these basic skills classes (which are non-transferrable) 
were targeted for study in order to demonstrate that students who score above a determined cut- 
off score perform more successfully in class than students who score below the cut-off score. 

The study was a post-hoc analysis of students' grades in the classes mentioned above for 
Fall 1990 and Spring 1991. In order for students to be admitted into the study, they had to meet 
the following criteria: 

4 Students had to have taken the math placement test on or after July 1, 1990, and 

a No intervening coursework was completed between the placement testing and 
enrolling in Math 050, 060, or 080. 

Students are considered successful if they received a grade of A,B,C, or Cr in the course. 
Students were deemed unsuccessful if they received a grade lower than a C or withdrew. 
Similarly, for the purposes of this study, "eligible" refers to students who met the cut-off criteria 
and "ineligible" defines those who did not meet that criteria. 

Once students were categorized as successful or unsuccessful, eligible and ineligible, and 
their test scores entered into the computer system, a two by two analysis using a crosstabs 
procedure was evoked in order to evaluate the findings. 

Further, in order to understand the correlation between the students' course outcome and 
placement test score, Pearson correlations were also computed. However, because of the limited 
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range which letter grades provide, it was felt that percentage scores were a better indicator of 
performance than was the letter grade for the correlation analysis. Students' total points in the 
class were converted, with the assistance of the instructors, to percentages so that we could 
control for the number of points possible in a course and also the number of tests. So if a 
person earned a total of 500 points out of a possible 800 in the course, they received a class 
performance score of 62.5%. This is the value that was correlated with their placement test 
score. 

Subjects 

After selecting students based on the above criteria, the final sample consisted of 51 % 
men and 49% women. The ethnic distribution was as follows: 

Table 1 



Ethnic Distribution of Sample 



Ethnicity 


Frequency 


White 


39% 


Hispanic 


36% 


Asian 


15% 


Black 


5% 


Filipino 


1% 


American Indian 


1% 


Pacific Islander 


1% 


Middle Eastern 


1% 


Other 


1% 



Test Materials 

Students entering Math 050 are currently required to score at least 2 1 points on the Math 
Placement Test #1 in order to be admitted into that course. Similarly, students entering Math 
060 have to have 12 points or greater on Math Placement Test #2 to be allowed entrance into 
that course. Students entering Math 080 must score 26 points or more on Math Placement Test 



#2. Because of the small group of students who took Math Placement Test #3, this study is 
focussing only on the validity of Math Placement Tests #1 and #2 as they pertain to these 
courses. 



Results 

Math 050 

It can be seen from Table 2 that of the 454 students in Math 050 in this study, 17% had 
scored below the cut score and 83% had scored above the cut score. Further, 47% of those 454 
students were successful in the course, and 53% were not. 

The success rate for students who scored above the cut score on the placement test was 
48%, and for those who scored below the cut score the success rate was seven percentage points 
less, 41%. 

The Pearson Correlation between the score on the Math Placement Test #1 and the 
performance score in the class was .33, p< .05. 

Given these two measures, it can be concluded that there is a positive relationship 
between the placement test score and course performance; students who score above the cut 
score are more often successful in the course. 



Table 2 

Comparison of Successful vs. Non-Successful 
Students Based on Test Performance 



Math 050 


Not 
Eligible 


Eligible 


ROW 
TOTAL 


Successful 


41% 


48% 


n=212 
47% 


Not 
Successful 


59% 


52% 


n=242 
53% 


COLUMN 
TOTAL 


n=78 
17% 


n=376 
83% 


n=454 



In order to evaluate the appropriateness of the current cut-off scores for Math 050, the 
sample was evaluated by hypothetically selecting out students who met different selection criteria 



Table 3 

Comparison of Student Performance Across Possible Cut Scores 



Score on RSC Math Placement Test #2 



MATH 050 
01=454) 


IS 


19 


20 


21 


22 


23 


24 


Not Eligible & 
Successful 


4% 
n=20 


6% 
n=25 


6% 
n=29 


7% 
n=32 


9% 
n=42 


11% 
n=48 


12% 
n=34 


Eligible & 
Successful 


42% 
n = 192 


41% 
n = 187 


40% 
n = 183 


40% 
n«180 


37% 
n = 170 


36% 
n = 164 


35% 
n = 158 


Not eligible & 
not successful 


5% 
n=24 


7% 
n=31 


9% 
n=41 


10% 
n=46 


15% 
n=69 


20% 
n=89 


24% 
n = 106 


Eligible & not 
successful 


48% 
n=218 


47% 
n=211 


44% 

n=201 


43% 
n=196 


38% 
n = 173 


34% 
n = 153 


30% 
n=136 



(different placement test scores). For example, in Table 3 it can be seen that using the current 
cut-off score of 21 tends to provide for an optimal number of students who are both eligible to 
take the course and who are, in fact, successful (40%). Inspection of the table suggests that if 
we were to increase the cut scores, we would be eliminating too many students who are, in fact, 
successful or, by decreasing the cut-score, allowing too many students into classes who are in 
greater risk of failing. 



Results 

Math 060 

Table 4 shows that 266 students were studied for Math 060, and 6% scored below the 
current cut-score and 94% scored above. Sixty-one percent (a higher success rate than for Math 
050) were successful in the course, and 39% were not. Those who were eligible for the course 
according to their placement test score had a success rate fifteen percentage points higher than 
that for ineligible students, 62% versus 47%. 

Table 4 

Comparison of Successful vs. Non-Successful 
Students Based on Test Performance 



Math 060 


Not 
Eligible 


Eligible 


ROW 
TOTAL 


Successful 


47% 


62% 


n = 163 
61% 


Not Successful 


53% 


38% 


n = 103 
39% 


COLUMN TOTAL 


n = 15 
6% 


n-251 
94% 


n=266 



These findings are commensurate with those of Math 050, inasmuch as students who are placed 
into the course by an appropriate placement test score are succeeding at a higher rate than 
students who score below the current cut-off scores on the test and are allowed into the course. 
The Pearson Correlation between the Math Placement Test #2 and course performarx .score for 
Math 060 was .14, c<.05. 

From Table 5 it can also be seen that hypothetically moving the cut-off score for the 
second Math Placement Test would jeopardize student entry into the program. For example, 
the current Placement Test #2 requirement is that a student score 12 or greater in order to be 
eligible for Math 060. However, if we artificially increased that cut-off score, we would be 
eliminating a number of students who would otherwise have been very successful. By the same 
token, if we lower the cut-off even one point, we would be allowing people into the course who 
would be at higher risk of failing. The results suggest that at this time there is no justification 
for changing the cut-off scores in this course. 
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Table 5 



Comparison of Student Performance Across Possible Cut Scores 



MATH 060 


Score on RSC Math Placement Test #2 


9 


10 


11 


12 


13 


14 


15 


Not Eligible & 
successful 


0% 
n=l 


0% 
n = l 


2% 
n=5 


3% 
n=7 


6% 
n = 17 


9% 
n=25 


13% 
n=34 


Eligible & 
Successful 


61% 
n = 162 


61% 
n=162 


59% 
n = 158 


59% 
n=156 


55% 
n=146 


52% 
n = 138 


49% 
n = 129 


Not EUgible & 
Not Successful 


1% 
n=2 


1% 
n=2 


1% 
n=3 


3% 
n=8 


7% 
n = 19 


9% 
n=24 


12% 
n=33 


EUgible & Not 
Successful 


38% 
n = 10i 


38% 
n= 101 


38% 
n = 100 


36% 
n=95 


32% 
n=84 


30% 
n=79 


26% 
n=70| 
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Results 
Math Q8Q 

When evaluating the results from Math 080 using Placement Test #2, we find that 37% 
of the students in the course were ineligible according to their placement test score versus 63% 
who were eligible; 63% were successful and 37% were not. The success rate for eligible 
students was 65% and for ineligible students it was 58%. 



Table 6 

Student Performance with Current Cut Score 





Not 
EUgible 


Eligible 


ROW 
TOTAL 


Successful 


58% 


65% 


n = 170 
63% 


Not 
Successful 


42% 


35% 


n = 101 
37% 


COLUMN 
TOTAL 


n = 101 
37% 


n = 170 
63% 


n=271 



This finding is consistent with the findings of other course samples, which points to the fact that 
students who were placed by the current cut-off criteria succeed at a higher rate. 

The Pearson Correlation between performance in the course and the test score #2 was 
.21, b<.05. 

As with the other courses, we artificially selected students at different cut scores for Math 
080, and from Table 7 we can see that the same type of decision would be made. We would 
eliminate too many students who would otherwise have been successful, and, at the same time 
if we lowered the test score cut-off at all, several students would be admitted who would fail. 
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Table 7 

Comparison of Student Performance Across Possible Cut Scores 



Score on RSC Math Placement Test #2 



MATH 080 
(n=169) 




24 


25 


26 


27 


28 


29 1 


Not Eligible & 
successful 


15% 


17% 
n — ho 


20% 

n -Cl 
11 —j J 


22% 


24% 

n<s=64 


27% 
n =74 

11 f *T 


31% I 
n =8S | 

11 %Jmf H 


Eligible & 
Successful 


48% 
n=130 


46% 
n=124 


43% 
n = 117 


41% 
n=lll 


39% 
n = 106 


35% 
n=96 


31% I 
n=85 1 


Not Eligible & 
Not Successful 


13% 
n=34 


14% 

n=38 


15% 
n=41 


16% 
n=42 


19% 
n=50 


20% 
n=55 


24% I 
n=64 I 


Eligible & Not 
Successful 


25% 
n = 67 


23% 
n=63 


22% 
n=60 


22% 
n=59 


19% 
n=51 


17% 
n=46 


14% | 
n=37 J 



Discussion of Findings 

The results of this evaluation show that the predictive validity of the Math Placement Test 
does, indeed, exist. There are increases in success rates in Oie .hree courses examined with 
respect to students who were eligible by test score to those who were not. 

it should be noted that one of the limitations of using this design is that students were 
ah oady placed into the course with a placement rule. Therefore, we do not have a representative 
sample of the full range of students' test scores or the students' course performance for those 
who scored below the cut-offs. Therefore, the success rate for the total number of students in 
the courses is quite high. For example, in Math 060, 61% of students are successful and in 
Math 080 63% are successful. 

If we were to test a random sample of students and then allow them to enter into Math 
050, we would expect that the majority of students who were eligible for the course would 
succeed at a much higher rate than those who were not eligible because the range of test scores 
in the course would be greater. 

The success rate for those who were ineligible for these courses are probably inflated 
because many could have been allowed into the class for other seemingly justifiable reasons. 
For example, counselors who advise students and note from their past experience that they have 
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been quite successful, or can determine that a student's motivation is quite high, may have let 
a student into a course based on those criteria even though they did not score above the cut-score 
required for a course. The success rates of the ineligible students in this study indicate that their 
placement was not always inappropriate, however. 

Another limitation of the current study is that anywhere from 12-21 % of the students who 
take Math 050, 060, and 080 do not have test scores; they were admitted into the course because 
they had completed a prerequisite course. Therefore, if we were able to test the entire sample 
of students in the Math Department, we might find that the results would be a little more robust. 

Yet another variable affecting the findings of this type of a study is the fact that students 
who received "W M grades were included in the sample studied. While it is true that some 
students withdraw because they do not have appropriate skills (those would be in the ineligible 
group), some withdraw for reasons other than those related to their preparatory skills. That is, 
it is possible that some of those categorized as unsuccessful by virtue of receiving a "W" grade 
could have been successful in the con ie had not the need to withdraw, for reasons other than 
lack of skills, presented itself. 

Su ggestions for Future Research 

Obviously, the best method for obtaining a predictor of validity is to conduct an 
experiment whereby a random sample of students is evaluated with a placement test all on the 
same day, and then are allowed to take any of the courses under question. Their performance 
can then be tracked during the semester and a much clearer evaluation of the impact of the test 
score can be shown. Given the fact that this type of research design creates an inordinate 
amount of administrative and classroom problems, below are some other suggestions offered for 
future study. From results of studies that we have done at Rancho Santiago College using 
reading test scores and student success in courses, the following recommendations are offered: 
4 A student's progress during the course should be monitored and correlated at 
distinct intervals of time. For example, during the semester the correlation 
between students' performance on major exams should be evaluated with their test 
score. In this way, several data points during the semester can be entered into 
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a multiple regression equation so that we have many more measures of the 
student's ability as opposed to just using their final course grade. 
All students could be tested on the first or second day of class in order to 
eliminate all the variables that come into the testing setting over a period of time. 
More stringent controls over the actual recording of the scores need to be put into 
place in order to avoid some of the problems that were encountered in the course 
of this study. 

Data related to instructor perception of student preparedness could be gathered 
early in the semester and compared with test scores. 



11 



• 



Addendum 



12 




5 



Addendum 



The richness of the data base created by the authors of this study has tempted this writer 
to conduct additional analyses beyond that initially planned and described in the foregoing. 
Further, it is the intention of RSC matriculation program managers to continually analyze the 
appropriateness of RSC's assessment mechanisms and to continually add to the work of test 
validation. The analyses of this report are also .mended to be used as examples for validation 
studies for other RSC courses and departments. 

Below are additional analyses conducted using the same data base as that used in the 
report under this cover. 



Math 050 



Performance in Math 050 


Eligible 
u=376 


Not EUgible j 
n=78 


>90% 


13% 


1% 


81-90% 


16% 


3% 


71-80% 


12% 


5% 


61-70% 


5% 


4% 


41-60% 


5% 


7% 


<41% 


5% 


3% 


none (W's, etc) 


44% 


77% 


TOTAL 


100% 


100% 






Course 


Average Math Placement Test Score 




Successful 


Non-Successful 


Math 050 (Test #1) 


27.67 


25.11 


B-.00 i 
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Course 


Average Final Course Grade I 


Eligible 


Not Eligible 


Math 050 


3.27 


3.37 


Math 060 


Performance in Math 060 


Eligible 
n=251 


Not EUgible 
n=15 


>90% 


17% 


7% 


81-90% 


23% 


7% 


7 J -80% 


17% 


26% 1 


61-70% 


11% 


13% 8 


41-60% 


6% 


7% 1 


<41% 


4% 


0% 1 


none (Ws, etc.) 


22% 


40% 


TOTAL 


100% 


100% 



Course 


Avenge Math Placement Test Score 




Successful 


Non-Successful 


Math 060 (Test #2) 


19.49 


17.14 


p=.oo 



Course 


Average Final Course Grade 


Eligible 


Not Eligible 


Math 060 


2.81 


3.33 
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Math 080 



Performance in Math 080 


Eligible 
n=170 


Not Eligible 
n=101 


>90% 


% 22% 


8% 


81-90% 


17% 


8% 


71-80% 


13% 


7% 


61-70% 


5% 


4% 


41-60% 


4% 


1% 1 


<41% 


1% 


0% | 


none (W's, etc.) 


38% 


72% | 


TOTAL 


100% 


100% ] 



Course 


Average Math Placement Test Score 




Successful 


Non-Si'ccessful 


Math 080 (Test #2) 


28.78 


25.55 


B=00 



Course 


Average Final Course Grade 


Eligible 


Not Eligible 


Math 080 


3.15 


3.04 



Conclusions 

a For each of the three courses studied, the average placement test score of 
successful students was significantly higher than that for non-successful students, 
indicating a positive relationship between course success and placement test score. 

a For each of the three courses studied, the percentage of students with W grades 
was higher for those students who were not eligible, by test score, for the course. 
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For each of the three courses studied, the percentage of students with a course 
performance score over 80% was much greater for students who had entered the 
course with a placement test score at or above the existing cut-score. 
The average final course grade for eligible and ineligible students does not 
distinguish between the two groups. 

These findings confirm those of the report: Students who score above the 
current placement test cut-scores for the three courses studied have a 
significantly higher probability of successfully completing the course. 
However, because the sample was limited by a placement rule already in 
effect, because some students in the courses who scored below the cut-score 
had been allowed into the course by a counselor according to other placement 
criteria, and because of the many students in the sample who received a " W" 
grade somewhat confound the findings, the success and performance score 
rates of those in this study who scored below the placement test cut-score 
were not terribly low. 
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